Entering Mathematical Equations Multimodally: Results on Usability and Interaction Patterns

Current interfaces for entering mathematical equations on computers are arguably limited and cumbersome. Mathematical notations have evolved to aid symbolic thinking and yet text-based interfaces relying on keyboard-and-mouse input do not take advantage of the natural two-dimensional aspects of mathematical equations. Due to its similarities to paper-based mathematics, pen-based handwriting input may be faster, more efficient, and more preferred for entering mathematics on computers. This paper presents an empirical usability study that tests this hypothesis. We also explored a multimodal input method combining handwriting and speech because we hypothesize it may aid both computer recognition and user cognition. Novice users were indeed faster in handwriting and enjoyed the handwriting modality more than a standard keyboard-and-mouse mathematics interface, especially as equation length and complexity increased. The multimodal handwriting-plus-speech method was faster and better liked than the keyboard-and-mouse method and was not much worse in these aspects than handwriting alone. In our domain, users’ selfcorrection of errors tends to be easier in handwriting than with the keyboard; however, the overall average error rate was slightly higher in these modalities. In addition, users’ speech while writing differed from when speaking alone. Finally, user errors in handwriting and speech were nonoverlapping, a fact which a multimodal recognition system could use to improve performance.